Minimizing System Correlation in SVM Training

1 Description of the problem We will consider a binary classiication task for which two separate classiiers are available. Each classiier may use diierent input features and diierent modeling techniques. In a setup like this, the nal decision is made based on a combination of the outputs generated by both classiiers with the hope that the nal performance will be better than the performance of the two individual classiiers. This, nevertheless, is not necessarily the case. In the extreme, if both classiiers were generating exactly the same output for each sample, the combined classiier could never have a better performance than the individual ones, independently of the combination procedure used. Intuitively, what we wish is to have two classiiers for which the within class correlation is small. This way, both classiiers contribute independent information leading to a better nal decision. In this work we will study the case in which one of the classiiers is given to us (we can consider this system as a black box which simply gives us a value for each sample) and the other one is an SVM which we need to train. Our goal is to modify the training criteria for the SVM so that the score resulting from this system is as little correlated as possible to the scores from the black box system. 2 Anti-correlation Kernel In this section we will derive the optimization problem we need to solve in order to achieve the combined goal of minimizing the error of the SVM system (which we will call S) while also minimizing the correlation of this system with the original black box system (which we will call B). Given a training set T = f(x We want to modify the objective function by adding a term 2 in the objective function, where is a tunable parameter and is the within-class correlation between system S and system B. Given the scores fb (i) ; i = 1; :::; mg from system B for the training set T, we can compute the within-class correlation between the scores produced by the SVM and these scores the following way: 2 = cov(B; SjY) 2 var(BjY)var(SjY) (2) where cov(B; SjY), var(BjY) and var(SjY) are the within-class covariance and variances. These can be approximated by the within-class sample covariance and variances in the training set T. The within-class sample covariance can be calculated as,